SelfCCL: Curriculum Contrastive Learning by Transferring Self-Taught Knowledge for Fine-Tuning BERT

BERT, the most popular deep learning language model, has yielded breakthrough results in various NLP tasks. However, semantic representation space learned by BERT property of anisotropy. Therefore, needs to be fine-tuned for certain downstream tasks such as Semantic Textual Similarity (STS). To overcome this problem and improve sentence space, some contrastive methods have been proposed fine-tuning BERT. existing models do not consider importance input triplets terms easy hard negatives during training. In paper, we propose SelfCCL: Curriculum Contrastive Learning model Transferring Self-taught Knowledge Fine-Tuning which mimics two ways that humans learn about world around them, namely curriculum learning. The former learns contrasting similar dissimilar samples. latter is inspired way from simplest concepts complex concepts. Our also performs training transferring self-taught knowledge. That is, figures out are or difficult based on previously knowledge, then those order using a objective. We apply our Sentence BERT(SBERT) frameworks. evaluation SelfCCL standard STS SentEval transfer show together with increases average performance extent.